Work in progress: comments welcome Evaluation in Multi-Site Programs

This paper discusses the problem of evaluating and predicting the treatment impact of a program that is implemented at multiple sites. Two issues arise: is information from other sites relevant in estimating the impact at a given site? and how can we account for predictive uncertainty in the site effects? Using data from the Greater Avenues for Independence evaluation, I develop a hierarchical model for earnings which allows both for site effects and for smoothing the estimated impact across sites. I show that the degree to which the estimated impact is smoothed across sites does not affect the estimate; i.e. most of the differences across sites are due to differences in the composition of participants. Second I show that predictive uncertainty regarding site effects is important; for example, when the predictive uncertainty regarding site effects is ignored, the treatment impact at the Riverside sites is significant, but when we consider predictive uncertainty, the impact for the Riverside sites is insignificant. Third, I show that the hierarchical model is able to extrapolate site effects with reasonable accuracy when the site for which the prediction is being made does not differ substantially from the sites already observed. For example, the San Diego treatment effects could have been predicted based on observable site characteristics, but the Riverside effects are consistently underestimated.